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Abstract — An immersive virtual reality display device such as an HMD (Head Mounted Display) provides virtual 
experiences by blocking external visual information and providing a computer generated virtual world. On the other hand, 
the augmented reality system does not present a complete virtual world as the immersive virtual reality system does, but it 
can further enhance the effect of reality by combining virtual objects in a specific space of the real world that the user sees. 
In this paper, we propose an immersive multi-view augmented reality system that can add advantages of immersive virtual 
reality system to augmented reality system. In our proposed system, in addition to the visual information of the camera 
attached to the HMD for the immersive environment, the visual information of the HMD camera was extended by showing 
the visual information captured by a mobile device in the same augmented reality space. 

Keywords — augmented reality, HMD (head mounted display), immersive, multi-view, virtual. 

I. Introduction 

With the recent popularization of smart mobile devices, digital contents of various genres can be conveniently used anywhere 
through wireless Internet. Especially, the spread of augmented reality and virtual reality service using smartphone and HMD 
is rapidly increasing. It can be said that augmented reality and virtual reality, which were only seen in movies and novels a 
few years ago, are becoming increasingly part of our lives. 

Virtual reality refers to a specific environment or artifact or technique itself created by artificial technology using a computer. 
A see-through HMD system, designed by Ivan Sutherland in the 1960s, is the first virtual reality system [1]. In the early 
days, the virtual reality system was not easy to succeed in actual and utilization because of the construction cost and lack of 
hardware and software technology. In recent years, however, with the development of remarkable hardware and software 
technologies, inexpensive commercialized products are beginning to be introduced. Commercial HMD technology allows 
users to experience environments or situations that could not be experienced in the real world. Examples include naval 
parachute training [4], rehabilitation therapy [5], and education [6]. However, because the virtual reality system is a 100% 
computer generated environment, it is not easy to apply it to applications based on objects that we can directly see and feel in 
the real world. 

On the other hand. Augmented Reality (AR), a technique derived from one field of virtual reality technology, cannot give 
users experience such as a 100% virtual environment or situation, but it can further increase the effect of reality by 
combining a virtual object with a specific space of the real world that the user sees. Thanks to these advantages, the 
augmented reality system can be applied to various fields such as personal information system [7], navigation [8], tourism 
[9], design [10], and medical treatment [11]. In recent years, augmented reality applications can run on mobile devices such 
as smart phones. On the other hand, an augmented reality system that can directly interact with a virtual object while viewing 
the real world has a disadvantage in that it can only show the real world ahead of users. 

In this study, to add the advantages of the virtual reality system to the augmented reality system, we implemented a system 
that shows the reality information in terms of multiple views. In addition to the visual information from the camera attached 
to the HMD, this system allows visual information from a mobile device to be displayed in the same augmented reality space, 
thereby extending the visual information of a single augmented reality camera. In addition, we propose a new augmented 
reality system by adding a convenient interface for switching between the views from multiple cameras. 

The structure of this paper is as follows. In Section 2, we discuss the augmented reality technology and HMD technology that 
are the background of this study. In Chapter 3, we describe the details of system configuration and implementation. Einally, 
Chapter 4 concludes this paper. 
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II. Research Background 

2.1 Augmented Reality Technologies 

Since the late 1990s, research on the Augmented Reality has been conducted mainly in the United States and Japan. In 1997, 
Ronald Azuma's definition of augmented reality was best explained in [12]. His definition is as follows. 1) Combination of 
real and imaginary images, 2) Capable of real-time interaction, 3) Placed in a three-dimensional space. 

In the point of experiencing a virtual world and interacting with users, it is similar to virtual reality, but it differs from virtual 
reality in that it can further enhance the effect of reality by combining a virtual object with a concrete space of the real world 
where users can see with their eyes. Thus, augmented reality emerged as an alternative to complement the virtual reality by 
embedding the insufficient attributes in the virtual reality. 

The augmented reality platform has evolved into mobile devices and smartphones since 2000, starting with desktop 
computers and workstations. The smartphone, which has become a necessity for modern people, has the most attention as the 
augmented reality platform because of its mobility and performance that is not lower than that of desktop computers. 
However, the smartphone -based augmented reality platform has the disadvantage of holding a smart phone in hand to view 
information. To overcome this problem, it is gradually evolving into eyeglass types [13]. 

According to the method of extracting feature points from the real image, the augmented reality systems can be classified 
into three classes as shown in Table 1. In the early days when it was difficult to apply the technology related to the camera 
vision in real time, augmented reality applications that depend only on sensors such as GPS, digital compass, accelerometer 
and gyroscope appeared. However, as the hardware and software technology developed, an augmented reality applications 
utilizing the camera vision technology emerged. Again, the camera vision based augmented reality can be classified into 
“Marker based AR” and “Marker-less AR” depending on whether the physical marker is required. "Marker based AR" is fast 
and accurate, but is sensitive to changes in lighting and has the hassle of attaching markers. On the other hand, "Marker-less 
AR" has the advantage of not needing a marker, but it requires a lot of computing resources for calculation. This research is 
based on marker-based augmented reality system which is easy to implement. 

Table 1 


Classification of Augmented F 

REALITY Systems 

Category 

Advantages 

Disadvantages 

Non- Vision based 
AR 

Easy to implement using sensors such as GPS, 
digital compass, accelerometer, and gyroscope 

Accurate location-based information enhancement 
is not possible 

Marker based AR 

High recognition rate and accuracy 

Sensitive to lighting changes and the need for 
markers 

Marker-less AR 

No marker required 

Very large computing resource requirements 


2.2 HMD Technologies 

Oculus VR, which can be said to have brought virtual reality to popularization, was first developed by Palmer Luckey who 
wanted to enjoy the game more realistically five years ago. Oculus Rift, released at the Game Developer Conference 2014, 
uses fisheye lenses to provide users with wide viewing angles. Because they could not raise the device’s size or extend the 
distance between the screen and eyes, the method of enlarging the area of the recognized screen using the lenses is applied. 
In other words, the problem of the screen size and the viewing angle is solved by making it possible to see a larger screen 
than the actual one while reducing the physical focal distance. However, the fisheye lens is subject to distortion in the optical 
structure. To solve this phenomenon, they use fisheye rendering technology to distort the image itself that is rendered on the 
screen. Thus, the image seen by the user is free from distortion. Oculus is equipped with an acceleration sensor, which 
changes the virtual reality point of view in front of user's eyes. Recently, the subsequent prototype "Oculus Crescent Bay" 
was released. The frame rate is improved compared to the previous type, and the sound changes according to the tracking 
direction thanks to the stereophonic system. 

Samsung Gear VR, which was created in cooperation with Oculus, utilizes the displays of the Galaxy Smartphone such as S6, 
S6 Edge, S7, S7 Edge and Note5, and provides wide viewing angle (96 degrees). This HMD has built-in accelerometer and 
gyro sensor, and it also supports a 360-degree panoramic view. The biggest advantage of this device is that it does not need 
to be connected to PC. 
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Other popular HMDs include Sony PS4VR, HTC Vive and Google Card Board. From this fact, we can see that many 
companies have a great interest in the field of virtual reality. Recently, there have been many cases of virtual reality 
applications where users can experience more active activities. Examples include a VR bicycle games [14], an exercise 
system [15], and a remote-controlled robot that allow robots to work safely in remote dangerous areas [16]. 

Microsoft has unveiled a device developed from another perspective, Hololens, at developer conference Build 2015. 
Hololens is based on augmented reality unlike the virtual reality based HMDs above. It is a type of goggle based on See- 
Through HMD. It can scan the 3d space where the user is located, and can be regarded as a wireless holographic computer 
that operates independently without a mobile phone or PC connection. This device recognizes objects in real-world space, 
augments 3D virtual images in the glasses’ screen, and can track user's eye movements and gestures. 

III. System Configuration and Implementation 
3.1 System Configuration 

In this study, we used Oculus DK2 as a device for presenting multiple views to users. We attached a webcam to the HMD to 
show the real image and transmitted the image of the smartphone to the PC and HMD using Unity network communication. 
In addition. Leap Motion [17] was used to switching between multiple images from 2 cameras by using hand gestures. 
Finally, we used Unity game engine to integrate Oculus, Leap Motion, a webcam, a smartphone, and augmented reality SDK. 
The overall configuration of the implemented system is shown in Fig. 1, and the actual usage screen is shown in Fig. 2. 


USB connection 



Fig. 1: Overall structure of the system 



Fig. 2: actual usage screen of the system 


3.2 HMD 

Oculus DK2, which is the most similar developer version to regular consumer products, embeds fisheye lenses and supports 
fisheye rendering to provide wide viewing angles. You can turn the dials on the left and right of the device to adjust the focal 
length to match users’ IPD (Inter-Pupillary Distance), and in the provided "Oculus Configuration Utility", you can fine-tune 
it to suit users’ physical characteristics. Since Oculus is not embedding a camera, we attached a webcam connected to a PC to 
the top of the HMD so that users could see the external environment. 
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The way to connect Oculus to Unity game engine is very simple. Importing the "Oculus Utilities" package from the Oculus 
homepage allows you to use a wide variety of Classes and Prefabs. In this project, the basic steps of preparation for using the 
HMD are finished by adding a Prefab called “OVRCameraRig” to the Hierarchy window of Unity game engine. 

3.3 Cameras and Multi-View Support 

In this study, there are two cameras used in image shooting. One is a webcam connected to the PC via USB and it is attached 
to the upper part of the HMD to show the image to be watched directly by the user, and the other one is the camera built in 
the smart device (using Google's NexusV in this study), which shows the near remote image rather than the image that is 
directly seen from the user's location. 


Table 2 

The four view modes in our system 


Mode 

Camera A 

Camera B 

Mode A. a 

Full Window 

X 

Mode A.b 

Full Window 

Sub Window 

Mode B.b 

X 

Full Window 

Mode B.a 

Sub Window 

Full Window 


We defined the four view modes in order to efficiently control the image viewed from the HMD (Camera A) and the one 
from the smart device (Camera B) (See Table 2). For example, in case of "Mode A.b", the image from "Camera A" is 
rendered to the whole screen, and the image from "Camera B" is rendered to the small sub window at the bottom of the 
screen. (See Figs. 3 to 6). 





3.4 Hand Gesture Recognition using Leap Motion 

Leap Motion is a controller released by a venture company in San Francisco. It is an input device that can be connected to a 
PC or a smart phone and lets you experience a virtual reality application by hand gestures. This device is very useful for 
variable virtual reality applications because it can track a user's hands, gestures, and finger bones. In this study, it was 
connected to the USB port of the HMD and attached in front of the HMD. 

To use Leap Motion in Unity game engine, you need to import the asset package provided on the Leap Motion website. This 
package includes some demo applications and various SDK modules. In this study, we did not make a new virtual hand but 
made some modifications to the one demo application (“Leap_Hands_Demo_VR”) in the asset package. 
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To switch between the four view modes, as described in section 3.3 above, we defined two hand gestures. To switch the main 
view (toggling between “Mode A.x” and “Mode B.x”), we use the gesture of right-hand from down to upward direction, and 
to toggle the display of the server view (toggling between “Mode X.a” and “Mode X.b”), we use the gesture of right-hand 
from left to right direction. 

3.5 Augmented Reality Implementation 

There are various SDKs such as Wikitude [18], ARToolKit [19], Vuforia [20], and Maxst [21] to help implement augmented 
reality systems. In this study, we used the Maxst SDK for faster image tracking because it can not only recognize 200 targets 
simultaneously, but also shows relatively robustness and fast results. To use Maxst in Unity engine, you can easily develop 
basic AR applications by importing the Unity package provided on the company’s homepage. In addition, to add a marker for 
recognition, you should use the service provided on the Maxst homepage. 

3.6 Unity Applications 

The Unity Applications of our system consists of a client program and a server program. The client runs on a mobile device 
such as a smart phone, and transmits the image obtained from the camera built in the device to the server, which is a 
composite image of the virtual object at the marker position found from the AR SDK. The server runs on the PC and 
transmits the image obtained from the Webcam connected to the PC and synthesized virtual objects to the marker position 
found from the Server's AR SDK to the HMD. Of course, the server also performs the work of combining and selecting the 
image generated by the server and the image transmitted from the client as described in subsections 3.4 and 3.5 above. 
Figure 7 shows the flow of video transmission between the server and the client. 





HMD 

Fig. 7: the flow of video transmission between the server and the client 

IV. Conclusion 

In this study, we implemented a multi-view augmented reality system using a mobile device. The proposed system provides a 
method to extend the user’s view by adding a remote augmented image to existing immersive video see-through type 
augmented reality system. In addition, we implemented hand gesture based interfaces for switching between the four views 
modes. 

The proposed system can be applied to a robot, an RC car or any network capable devices, and it is also possible to connect 
multiple devices at the same time. This enables augmented reality-based tasks without going directly to physically separated 
(or dangerous) areas, and can be applied to areas where multi-user collaboration is required. 
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